An Empirical and Formal Analysis of Decision Trees for Ranking

Decision trees are known to be good classifiers but less good rankers. A few methods have been proposed to improve their performance in terms of AUC, along with first empirical evidence showing their effectiveness. The goal of this paper is twofold. First, by replicating and extending previous empirical studies, we not only improve the understanding of earlier results but also correct implicit assumptions and conjectures. We focus on the dependence of AUC on the pruning level, the effect of Laplace correcting the class frequencies in leaf nodes, and the influence of the number of distinct scores produced by a decision tree. Second, we complement the empirical studies by a formal analysis that offers further insights and explanations. Essentially, we show that the AUC is likely to increase with the number of scores produced by a tree, at least when these scores are reasonable (better than random) approximations of the true conditional probabilities in the leaves. Simulation studies with synthetic data verify our formal analysis and show its robustness toward estimation errors. As a byproduct, our result suggests a simple method for improving the ranking performance of decision trees, and it helps to understand why some classifiers are better rankers than others.